#crawler controls

Cloudflare replaces its blanket AI bot block with granular controls for search, training, and agent crawlers

Cloudflare is introducing granular AI bot controls, allowing website owners to manage Search, Training, and Agent bots separately. Starting September 15, 2026, Training and Agent bots will be blocked by default on ad-supported pages.